Control apparatus, image pickup apparatus, control method, and storage medium

ABSTRACT

A control apparatus includes a first processing unit configured to calculate correlation information between image signals, which are respectively output from one of the pixel portions positioned in one of a plurality of first regions, a focus detection unit configured to perform focus detection on the basis of a calculation result by the first processing unit, a second processing unit configured to calculate correlation information between image signals, which are respectively output from the one of the pixel portions positioned in one of a plurality of second regions, a distribution information calculation unit configured to calculate distribution information corresponding to an object distance on the basis of a calculation result by the second processing unit, and a control unit configured to control the second processing unit and the distribution information calculation unit on the basis of the correlation information calculated by the first processing unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image pickup apparatus, its controlapparatus, and its control method.

2. Description of the Related Art

In some image pickup apparatuses such as a digital camera and a digitalvideo camera, pixels of the whole or apart of an image sensor havedistance measuring function. Such image pickup apparatus includes aplurality of photoelectric conversion portions for each pixel, anddetects a distance to an object by a phase-difference method. Forexample, Japanese Patent Laid-Open No. 2014-74891 describes an imagepickup apparatus that generates an image and distance distributioncorresponding to the image (hereinafter referred to as “distance map”)using the above mentioned technique.

Generating a distance map, however, requires a correlation calculationof a focus detection signal obtained from each photoelectric conversionportion for each pixel, and therefore the calculation load on acalculation circuit increases. It has a problem that processing of acalculation circuit is delayed, etc.

SUMMARY OF THE INVENTION

This invention provides an image pickup apparatus capable of reducingthe load on a calculation circuit when generating a distance map, itscontrol apparatus, and its control method.

A control apparatus, which is used for an image pickup apparatusincluding an image sensor that has a plurality of pixel portionsrespectively having a plurality of photoelectric conversion portions andoutputs an image signal from the photoelectric conversion portions, asone aspect of the present invention includes a first processing unitconfigured to calculate correlation information between image signals,which are respectively output from one of the pixel portions positionedin one of a plurality of first regions, a focus detection unitconfigured to perform focus detection on the basis of a calculationresult by the first processing unit, a second processing unit configuredto calculate correlation information between image signals, which arerespectively output from the one of the pixel portions positioned in oneof a plurality of second regions, wherein the one of the second regionsincludes a region near the one of the first regions, is larger than theone of the first regions, and corresponds to the one of the firstregions, a distribution information calculation unit configured tocalculate distribution information corresponding to an object distanceon the basis of a calculation result by the second processing unit, anda control unit configured to control the second processing unit and thedistribution information calculation unit on the basis of thecorrelation information calculated by the first processing unit.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image pickup apparatus in thisembodiment.

FIG. 2 is a pixel layout diagram of an image sensor in this embodiment.

FIG. 3 is a diagram that illustrates the relationship of an imaging lensand a unit pixel portion in this embodiment.

FIG. 4 is a schematic diagram of the whole structure of an image sensorin this embodiment.

FIG. 5 is a diagram that illustrates the structure of a unit pixelportion in this embodiment.

FIG. 6 is a diagram that illustrates the structure of a column commonreadout circuit in this embodiment.

FIG. 7 is a function block diagram of DSP in this embodiment.

FIG. 8A to 8D are diagrams that illustrate an image data arrangement, afocus detection area, and a distance map area.

FIG. 9A to 9C are diagrams that illustrate the structure of correlationdata in a focus detection area in this embodiment.

FIGS. 10A and 10B are flowcharts of a control in this embodiment.

FIG. 11A is a diagram for explaining calculation processing of N framein this embodiment.

FIG. 11B is a diagram for explaining calculation processing of N+1 framein this embodiment.

FIG. 11C is a diagram for explaining calculation processing of N+2 framein this embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the attached drawings. FIG. 1 is a diagramillustrating the whole structure of an image pickup apparatus 100 inthis embodiment.

A first lens unit 101 configures an image pickup optical system, and isarranged at a front end part of a lens barrel (object side) and is heldto be capable of being advanced and retreated along an optical axisdirection. A diaphragm 102 adjusts a light amount in photographing byadjusting its aperture diameter. A second lens unit 103 varies themagnification (zoom function) along with the advance and retreatoperation of the first lens unit 101. A third lens unit 104 is afocusing lens (focus lens), and performs a focusing by advance andretreat in the optical axis direction. An optical low pass filter 105 isan optical element for reducing false color or moire of taken images.

An image sensor 106 performs a photoelectric conversion of an objectimage formed by an image pickup optical system. In this embodiment, theimage sensor 106 utilizes a CMOS image sensor having Bayer arrangement.Each pixel in the image sensor 106 includes a plurality of photoelectricconversion portions. Specifically, a plurality of photodiodes(hereinafter simply referred to as “PD”) is disposed in a pixel. Thenumber of PD in a pixel portion is two or more (two, four, nine, etc.),and in this embodiment two PDs form sub pixel a and sub pixel b. Asignal used for focus detection (focus detection signal) and a signalused for image generation (image generation signal) are obtained fromthe sub pixels.

AFE (Analog Front End) 107 converts, to digital signal, analog signaloutputted from the image sensor 106. DFE (Digital Front End) 108performs calculation processing as preprocessing to the digital imagesignal outputted from the AFE 107. DSP (Digital Signal Processer) 109performs correction processing, developing processing, or the like forthe digital image signal outputted from the DFE 108.

A storage medium 110 stores image data processed by the DSP109. Adisplay portion 111 is configured from a liquid crystal display (LCD)displaying a taken image, a various menu screen, etc., or the like. RAM(Random Access Memory) 112 is a device that temporarily stores imagedata etc., and is connected with the DSP 109. A timing generator (TG)113 supplies a driving signal to the image sensor 106.

CPU (Central Processing Unit) 114 controls the AFE107, the DFE108, theDSP109, the TG113, the diaphragm, and a driving circuit 115.Furthermore, CPU 114 performs an AF (autofocus) control. Specifically,the CPU 114 calculates a defocus amount from a correlation calculationresult calculated by the DSP 109, and controls a focus driving circuit116 according to the defocus amount. The focus driving circuit 116 is acircuit that drives a focus actuator 118. The focus actuator 118advances and retreats the third lens unit 104, thereby performingfocusing. A diaphragm driving circuit 115 controls a diaphragm actuator117 according to the control instruction of the CPU 114, thereby drivingthe diaphragm 102. ROM (Read Only Memory) 119 stores correction data orthe like. A control portion 120 includes a shutter switch button (SW). Auser performs an operation such as half pressing and full pressing tothe shutter switch button, and an instruction signal is input to the CPU114. Moreover, the control portion 120 includes a dial, a touch panel,etc., and enables a various operation in shooting modes, setting ofvarious shooting parameters, reproducing modes, menu screens, or thelike.

FIG. 2 is a schematic diagram that illustrates a pixel layout example ofthe image sensor 106. A unit pixel portion 300 is arranged in matrix,and color filters of R (Red)/G (Green)/B (Blue) in unit pixel portion300 are arranged in Bayer array. The sub pixel a and sub pixel b arearranged in each unit pixel portion 300. In FIG. 2, PD 401 a is a firstphotoelectric conversion portion that forms the sub pixel a, and PD 401b is a second photoelectric conversion portion that forms the sub pixelb. Each signal from the sub pixel a and the sub pixel b is also utilizedfor focus detection or distance information to an object.

FIG. 3 is a diagram that illustrates the relationship between the unitpixel portion 300 and light flux emitted from an exit pupil 203 of animaging lens composed of the first lens unit 101, the diaphragm 102, thesecond lens unit 103, and the third lens unit 104. As described above,the unit pixel portion 300 includes PD 401 a and PD 401 b. A colorfilter 201 and a microlens 202 are arranged between the unit pixelportion 300 and the exit pupil 203.

An optical axis 204 indicates the center of light flux emitted from theexit pupil 203 toward a pixel portion having the microlens 202. Lightthat passes the exit pupil 203 enters the unit pixel portion 300 withthe optical axis 204 as a center. Each of areas 205 and 206 represents apart area of the exit pupil 203 of the imaging lens. As illustrated inFIG. 3, light flux that has passed the area 205 passes through themicrolens 202 and is received on the PD 401 a (sub pixel a). Light fluxthat has passed the area 206 passes through the microlens 202 and isreceived on the PD 401 b (sub pixel b). In other words, each of the subpixel a and the sub pixel b receives light that has passed differentarea of the exit pupil 203 of the imaging lens. Comparing output signalsfrom the sub pixel a and the sub pixel b enables focus detection by thephase-difference method.

When a state is out of the in-focus state, image signal waveformsrespectively obtained from sub pixels a, b are greatly shifted. As itapproaches the in-focus state, the shift between the image signalwaveforms decreases. In the in-focus state, both waveforms overlap witheach other. By utilizing such a phenomenon, the defocus amount can bedetected based on the shift amount of each image signal waveformobtained from the sub pixels a, b.

Next, the structure of the image sensor 106 will be described withreference to FIGS. 4 to 6. FIG. 4 is a diagram that illustrates thewhole structure of the image sensor 106. FIG. 5 is a circuit diagramthat illustrates the structure of the unit pixel portion 300. FIG. 6 isa circuit diagram that illustrates the structure of the column commonreadout circuit 303.

Many unit pixel portions 300 (see p11˜pkn) are arranged in matrix in apixel area illustrated in FIG. 4.

The PDs 401 a, 401 b perform the photoelectric conversion of incidentlight, and accumulate charge according to exposure amount. Transfergates 402 a, 402 b turn the ON state by setting respective signals txa,txb of the transfer gates 402 a, 402 b to High level. As a result,charge accumulated in PDs 401 a, 401 b is transferred to an FD (floatingdiffusion) part 403. The FD part 403 is connected to a gate floatingdiffusion amplifier 404 (hereinafter referred to as “FD amplifier”). AnFD amplifier 404 converts change amount, transferred from the PDs 401 a,401 b, into voltage amount. An FD reset switch 405 resets the FD part403 by setting a signal res to High level. When charge in PDs 401 a, 401b is reset, the signal res and the signals txa, txb are simultaneouslyset to High level. When the transfer gates 402 a and 402 b and the FDreset switch 405 turn the ON state, the PDs 401 a, 401 b is reset viathe FD part 403. A pixel select switch 406 changes signal sel to Highlevel, and a pixel signal converted into a voltage by the FD amplifier404 is output from an output terminal vout of the unit pixel portion300.

A vertical scanning circuit 301 in FIG. 4 supplies gate control signal(res, txa, txb, sel) to a transistor of the unit pixel portion 300. Thecontrol for the signals is common in each row. The output terminal voutof each unit pixel portion 300 is connected to the column common readoutcircuit 303 via a vertical output line 302 in each column. The structureof the column common readout circuit 303 will be described below withreference to FIG. 5.

The vertical output line 302 is set in each column, and is connected tothe output terminal vout of the unit pixel portions 300 in one column.To the vertical output line 302, a current source 304 is connected. Thiscurrent source 304 and the FD amplifier 404 of the unit pixel portion300 connected to the vertical output line 302 constitute a sourcefollower circuit.

A clamp capacitance (C1) 501 is connected to an inverted input terminalof a calculation amplifier 503. A clamp capacitance (C2) 502 isconnected to an output terminal and an inverted input terminal of thecalculation amplifier 503. To the non-inverted input terminal of thecalculation amplifier 503, a reference power Vref is connected. A switch504 is a transistor for causing short circuit between both terminals ofa feedback capacitance C2, and is controlled by a signal cfs. Each oftransfer switches 505 to 508 is a transistor for transferring a signalreadout from the unit pixel portion 300 to each of signal holdingcapacitors 509 to 512. By a readout operation described below, a pixelsignal Sa of a sub pixel a is memorized in a first S signal holdingcapacitor 509, and the addition signal Sab obtained by adding a signalof a sub pixel a and a signal of a sub pixel b is memorized in a secondS signal holding capacitor 511. Moreover, noise signals N of the unitpixel portion 300 are stored in a first N signal holding capacitor 510and a second N signal holding capacitor 512. The signal holdingcapacitors 509 to 512 are respectively connected to output terminalsvsa, vna, vsb, vnb of the column common readout circuit 303.

The output terminals vsa, vna of the column common readout circuit 303in FIG. 4 are respectively connected to the horizontal transfer switches305, 306. The horizontal transfer switches 305, 306 are controlled by anoutput signal ha* (* denotes arbitrary column number) of a horizontalscanning circuit 311. If the signal ha* is set to High level, signals inthe first S signal holding capacitor 509 and the first N signal holdingcapacitor 510 are respectively transferred to the horizontal outputlines 309, 310.

Moreover, horizontal transfer switches 307, 308 are respectivelyconnected to the output terminals vsb, vnb of the column common readoutcircuit 303. The horizontal transfer switches 307, 308 are controlled byan output signal hb* (* denotes arbitrary column number) of thehorizontal scanning circuit 311. If the signal hb* is set to High level,signals in the second S signal holding capacitor 511 and the second Nsignal holding capacitor 512 are respectively transferred to thehorizontal output lines 309, 310.

The horizontal output lines 309, 310 is respectively connected to inputterminals of differential amplifier 314. The differential amplifier 314calculates the difference of an S signal and an N signal, applies apredetermined gain to it, and then outputs its final output signal to anoutput terminal 315. Horizontal output line reset switches 312, 313 turnthe ON state by setting a signal chres to High level, and each of thehorizontal output lines 309, 310 is set to a reset voltage Vchres(reset).

Next, functional blocks of the DSP109 in this embodiment will bedescribed with reference to FIGS. 7 to 9.

The image data corresponding to the sub pixel a, obtained from the groupof unit pixel portions 300, denotes L(X,Y) and the image datacorresponding to the sub pixel b denotes R(X, Y) (X, Y are an integernumber more than 0, and indicate the coordinates of a pixel position).In this case, a/b addition signal data (hereinafter simply referred toas “addition data”) N (X, Y) is represented by the following expression:

N(X,Y)=L(X,Y)+R(X,Y)  (1)

In other words, the addition data N (X, Y) is also image datacorresponding to each group of unit pixel portions 300.

In embodiment, the image sensor 106 outputs R(X, Y) and N(X, Y) as oneset. In addition, a set of image data output from the image sensor 106may be a set of other data. For example, a set of L(X, Y) and N(X, Y) ora set of R(X, Y) and L(X, Y) is output.

The addition data N (X, Y) and the image data R(X, Y) output from theimage sensor 106 are input to the DSP 109 through the AFE 107 and theDFE 108.

FIG. 8A illustrates the addition data N (X, Y) and FIG. 8B illustratesthe image data R (X, Y). In this embodiment, as illustrated as FIGS. 8Aand 8B, each data is divided into seven parts in a horizontal directionand five parts in a vertical direction. The division number is notlimited to this.

As illustrated as FIG. 7, the addition data N (X, Y) and the image dataR (X, Y) are input to a first correlation calculation part 602 and asecond correlation calculation part 604. Moreover, the addition data N(X, Y) processed by signal processing and compression using a whitebalance circuit, a color interpolation circuit, noise reductionprocessing, gamma processing, matrix processing, degenerationprocessing, which are known, is output to the display portion 111 andstorage medium 110.

The first correlation calculation part 602 calculates the image data L(X, Y) by subtracting the image data R (X, Y) from the addition data N(X, Y) in a focus detection region (first region) set by the CPU 114.The first correlation calculation part 602 calculates correlation dataAfd (X, Y) (correlation information) of the calculated image data L (X,Y) and the image data R (X, Y) in a region set by the CPU114 on thebasis of the calculated image data L (X, Y) and the image data R (X, Y)using the following method. In this embodiment, the CPU 114 selects atleast a part of data as illustrated as FIG. 8C and sets a focusdetection region in which correlation data Afd (X, Y) is calculated. Forexample, when a user sets an AF control with respect to a center of datathrough the control portion 120, the CPU 114 sets a region obtainedcorrelation data Afd (3, 2) of FIG. 8C as a focus detection region andperforms correlation calculation using only signal in the region.However, when a mode that generates distance distribution information isset as a shooting mode, correlation calculation of a plurality of focusdetection regions is performed so as to obtain a plurality ofdistribution information regardless of a set AF mode (set of a focusdetection region).

The first correlation calculation part 602 calculates correlation dataAfd (X, Y) by comparing signal waveforms of the image data L (X, Y) andthe image data R (X, Y) while both data are shifted in a horizontaldirection. Here, when a shift amount is set to a minus when the imagedata L (X, Y) is shifted to the left and to a plus when the image data R(X, Y) is shifted to the right, shift movement from −S to S isperformed.

The first correlation calculation part 602 compares signal waveforms ofboth image data in a horizontal direction for each shift movement.Specifically, the first correlation calculation part 602 calculates avalue added one line value of the smaller of the image data L (X, Y) andthe image data R (X, Y) in each position as an one line correlationvalue for each shift amount. And, a correlation value of the same focusdetection region for each shift amount is derived by adding an one linecorrelation value of each line in the same focus detection region foreach shift amount and is input to a control part 600 as correlation dataAfd (X, Y). A correlation value in each focus detection region becomesmaximum during being a shift amount that signal waveforms of the imagedata L (X, Y) and the image data R (X, Y) overlap (approach). Acalculation method of correlation data is not limited to this, and maybe a calculation method indicating correlation of the image data L (X,Y) and the image data R (X, Y).

FIG. 9A to 9C are diagrams that illustrate the structure of correlationdata Afd (X, Y) input to the control part 600. The controller 600 storesa correlation value of each shift amount (−S to S) calculated by thefirst correlation calculation part 602 in the RAM 112 for each Afd (X,Y). RAM 112 stores at least two frame correlation data. Correlation dataof the control part 600 and two frame correlation data are compared, anda calculation operation by the second correlation calculation part 604is controlled on the basis of the comparison result using the followingmethod.

The second correlation calculation part 604 calculates the image data L(X, Y) by subtracting the image data R (X, Y) from the addition data N(X, Y) as the first correlation calculation part 602. The secondcorrelation calculation part 604 calculates correlation data Defs (X, Y)in a region set by the CPU114 on the basis of the calculated image dataL (X, Y) and the image data R (X, Y) using the same calculation methodthat the second correlation calculation part 604 calculates correlationdata Afd (X, Y). In this embodiment, the CPU 114 sets a distance mapregion in which correlation data Defs (X, Y) is calculated asillustrated as FIG. 8D. The distance map region is a region in which theCPU114 actually calculates a distance map on the basis of correlationdata. In this embodiment, the distance map is the entire region of imagedata but is not limited to this. When a focus detection region isnarrower than the distance map, a load of calculation processing can bereduced. Specifically, when the second correlation calculation part 604stops or performs calculation processing of correlation data Defs (X, Y)according to instructions from the control part 600, a load ofcalculation processing can be reduced. The second correlationcalculation part 604 may stop calculation processing of the image data L(X, Y) according to instructions from the control part 600.

Correlation data Defs (X, Y), which is calculated by the secondcorrelation calculation part 604 processed by signal processing andcompression using a white balance circuit, a color interpolationcircuit, noise reduction processing, gamma processing, matrixprocessing, degeneration processing, which are known, is stored asdistance map (object distance distribution) including distanceinformation to an object in the storage medium 110. Moreover, the secondcorrelation calculation part 604 may calculate a defocus amount and theCPU 114 (distance calculation part) may calculate a defocus amount fromcorrelation data Defs (X, Y). Additionally, the calculated correlationdata Defs (X, Y) (image shift amount) and the calculated defocus amountmay be directly stored as distance map in the storage medium 110. Inother words, in this embodiment, information indicating distribution ofinformation corresponding to an object distance (a distance to a depthdirection in the image data) may be obtained, and the obtainment methodof the information is not limited to the above method.

Next, a control of the DSP 109 by the CPU 114 according to thisembodiment will be explained referring to flowcharts in FIGS. 10A and10B.

FIG. 10A illustrates a main control performed in the DSP 109. Thecontrol in FIG. 10A is performed not only in a shooting mode, in which astill image and or a moving image is just captured and stored but alsoin a shooting mode, in which a moving image is captured and stored, andobject distribution information is generated. Obtaining the image data L(X, Y) and R (X, Y) and calculating distribution information using acorrelation calculation with respect to the entire screen always obtaindistribution information corresponding to one object distance relativeto each frame (or a plurality of frames), but the calculation has a highload and largely occupies a band of a bus. In this embodiment, acalculation of distribution information in each region (second region)in a screen is controlled on the basis of a result of correlationcalculation for AF calculated in a focus detection region. Thus, anunnecessary update of distribution information decreases and a load of acalculation reduces.

At step S1001, the DSP 109 determines whether or not calculationprocessing to the second correlation calculation part 604 is instructed.The flow proceeds to step S1002 when calculation processing isinstructed, and the flow proceeds to step S1003 when calculationprocessing is not instructed.

At the step S1002, the DSP 109 performs the correlation processing atthe first correlation calculation part 602 and the second correlationcalculation part 604 using the addition data N (X, Y) and the image data(X, Y). Meanwhile, at the step S1003, the DSP 109 performs thecorrelation processing only at the first correlation calculation part602 using the addition data N (X, Y) and the image data (X, Y) and stopsthe correlation processing at the second correlation calculation part604.

At step S1004, the DSP 109 (control part 600) determines whether or notthe calculation processing of the second correlation calculation part604 at a next frame is performed. A branch selected at the step S1001 ischanged according the determination at this step.

Next, specific processing performed by the control part 600 will beexplained referring to FIG. 10B.

At step S2001, the control part 600 obtains correlation data Afd (X, Y)from the first correlation calculation part 602 and stores it to the RAM112. At step S2002, the control part 600 determines whether a defocusamount is larger than a threshold value set by the CPU 114 in all focusdetection region on the basis of the correlation data Afd (X, Y). Theflow proceeds to step S2003 when the defocus amount is smaller than thethreshold value in all focus detection region, and the flow proceeds tostep S2005 when the defocus amount is larger than the threshold value inall focus detection region. The threshold value is set by the CPU 114.

At Step S2003, the control part 600 reads out correlation data Afd (X,Y) of a current frame (present) at the step S2001 and correlation dataAfd of a previous frame (past) from the RAM 112 to compare them.Specifically, comparison data Comp (X, Y) being an absolute value ofdifferences between defocus amounts at both frames is obtained from theAfd (X, Y) of the current frame, and it is determined whether or not thecomparison data is larger than a threshold value in all focus detectionregion. The flow proceeds to step S2004 when the comparison data islarger than the threshold value in all focus detection region, and theflow proceeds to step S2005 when the comparison data is smaller than (oris equal to) the threshold value in all focus detection region. Thethreshold value is set by the CPU 114. Moreover, the comparison dataComp (X, Y) may be a comparison result between the absolute value ofdifferences between defocus amounts and the threshold value.

At Step S2004, the control part 600 instructs the second correlationcalculation part 604 to perform correlation processing at next frame,and ends the processing at the current frame.

At Step S2005, the control part 600 instructs the second correlationcalculation part 604 to stop correlation processing at next frame, andends the processing at the current frame.

According to the determination at the step S2002, when image blurobtained from the image sensor 106 is large, stopping calculationprocessing of correlation data Defs (X, Y) reduces a load of thecalculation circuit. Additionally, according to the determination at thestep S2003, when variation of a defocus amount of an image obtained fromthe image sensor 106, stopping calculation processing of correlationdata Defs (X, Y) reduces a load of the calculation circuit.

Next, processing of the DSP 109 during capturing a moving image aftercorrecting image blur to a certain extent corresponding to steps fromthe step S1005 to the step S1008 will be specifically explainedreferring to FIGS. 11A, 11B, and 11C. FIGS. 11A to 11C illustrate anoperation of the DSP 109 during capturing a moving image. Frames N toN+2 respectively represent a frame number of a moving image, and FIGS.11A to 11C respectively illustrate processing at the N frame to the N+2frame. Under the frame number, input numbers input to the DSP 109 fromthe DFE 108 and correlation data Afd (X, Y) output from the firstcorrelation calculation part 602 are represented. Further, under them,comparison data Comp (X, Y) obtained by the control part 600 andcorrelation data Defs (X, Y) output from the correlation calculationpart 604 are represented. Moreover, in the following descriptions, thecontrol part 600 previously includes correlation data Afd (X, Y) of eachshift amount (−S to S) calculated from an image of the N−1 frame.

First, at the N frame (image frame N), processing to stop calculationprocessing of correlation data Defs (X, Y) of the second correlationcalculation part 604 at the N+1 frame will be explained referring toFIG. 11A. A digital image signal of the N frame is input, in order froma top line, to the DSP 109 from the DFE 108. Then, both the additiondata N (X, Y) and the image data R (X, Y) are input to the DSP 109.

The second correlation calculation part 604 starts the correlationcalculation processing at a timing where the addition data N (0, 0) andthe image data R (0, 0) are input. In the correlation calculationprocessing, the Defs (X, Y) is calculated according to the above method.And, the second correlation calculation part 604 outputs correlationcalculation data Defs (X, Y) in a distance map region in order fromcorrelation data Defs (0, 0) after a correlation calculation processingtime t, and ends the correlation calculation processing at a timingwhere correlation data Defs (6, 4) is output.

The first correlation calculation part 602 starts the correlationcalculation processing at a timing where the addition data N (1, 1) andthe image data R (1, 1) are input. In the correlation calculationprocessing, the Afd (X, Y) is calculated according to the above method.And, the first correlation calculation part 602 outputs correlationcalculation data Afd (X, Y) in a focus detection region in order fromcorrelation data Afd (1, 1) after a correlation calculation processingtime t, and ends the correlation calculation processing at a timingwhere a correlation value of calculation data Afd (5, 3) in a focusdetection region is output.

The control part 600 starts storage of correlation data Afd (X, Y) tothe RAM 112 at a timing where the first correlation calculation part 602inputs correlation data Afd (1, 1).

And the control part 600 starts a comparison operation between the N−1frame and the N frame relative to correlation data Afd (X, Y) after theaddition data N (6, 4) and the image data R (6, 4), which are the lastdata of the N−1 frame, are input to the DSP 109. The control part 600,in the comparison operation, determines whether or not an absolute valueof differences between correlation data Afd (X, Y) of the preceding N−1frame and the N frame is larger than a threshold value set by the CPU114.

When checking that all comparison data Comp (X, Y) are larger than thethreshold value, the control part 600 performs a selection operation Selfor next frame processing. In the selection operation Sel, flames usedfor the comparison operation are set to the N frame and the N+1 frame,and a region to store correlation data Afd (X, Y) of the N+1 frame isreserved in the RAM 112.

The control part 600 performs a STOP operation to stop the correlationprocessing of correlation data Defs (X, Y) by the second correlationcalculation part 604 at a timing where the N frame ends.

Next, processing of the DSP 109 during stopping the correlationcalculation processing by the second correlation calculation part 604 atthe N+1 frame will be explained referring to FIG. 11B. The explanationof the same processing as the N frame is omitted.

The control part 600 starts storage of correlation data Afd (X, Y) tothe RAM 112 at the N+1 frame at a timing where correlation data Afd(1, 1) is input by the first correlation calculation part 602. And thecontrol part 600 stars a comparison operation between the N frame andthe N+1 frame relative to correlation data Afd (X, Y) after the additiondata N (6, 4) and the image data R (6, 4), which are the last data ofthe N+1 frame, are input to the DSP 109.

After completing the comparison operation, the control part 600 performsthe selection operation Sel for next frame processing. In the selectionoperation Sel, flames used for the comparison operation are set to theN+1 frame and the N+2 frame, and a region to store correlation data Afd(X, Y) of the N+2 frame is reserved in the RAM 112.

Next, processing by the DSP 109 while the calculation performing ofcorrelation data Defs (X, Y) by the second correlation calculation part604 is restored from a stop state at the N+2 frame referring to FIG.11C. The explanation of the same processing as the N frame is omitted.

The control part 600 starts storage of correlation data Afd (X, Y) tothe RAM 112 at the N+2 frame at a timing where correlation data Afd(1, 1) is input by the first correlation calculation part 602. And thecontrol part 600 stars a comparison operation between the N+1 frame andthe N+2 frame relative to correlation data Afd (1, 1) after the additiondata N (6, 4) and the image data R (6, 4), which are the last data ofthe N+2 frame, are input.

After completing the comparison operation, the control part 600 performsthe selection operation Sel for next frame processing. Comparison dataComp (4, 2) exceeds the threshold value at the N+2 frame according tothis embodiment.

In the selection operation Sel, flames used for the comparison operationare set to the N+2 frame and the N+3 frame, and a region to storecorrelation data Afd (X, Y) of the N+3 frame is reserved in the RAM 112.

The control part 600 performs an Exe operation to execute calculationprocessing of correlation data Defs (X, Y) by the second correlationcalculation part 604 at a timing where the N+2 frame ends.

Thus, determining whether or not a distance map is calculated at nextframe according to image data and decreasing calculation operation withless importance reduce a load of the calculation circuit. When a defocusamount and comparison data respectively are larger than a thresholdvalue in all focus detection region, calculation of distance map isstopped at next frame. However, when they exceed the threshold value inapart of focus detection region, calculation of distance mapcorresponding to next frame may be stopped.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-026989, filed on Feb. 13, 2015, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. A control apparatus for controlling an imagepickup apparatus including an image sensor that has a plurality of pixelportions respectively having a plurality of photoelectric conversionportions and outputs an image signal from the photoelectric conversionportions, the control apparatus comprising: a first processing unitconfigured to calculate correlation information between image signals,which are respectively output from one of the pixel portions positionedin one of a plurality of first regions; a focus detection unitconfigured to perform focus detection on the basis of a calculationresult by the first processing unit; a second processing unit configuredto calculate correlation information between image signals, which arerespectively output from the one of the pixel portions positioned in oneof a plurality of second regions, wherein the one of the second regionsincludes a region near the one of the first regions, is larger than theone of the first regions, and corresponds to the one of the firstregions; a distribution information calculation unit configured tocalculate distribution information corresponding to an object distanceon the basis of a calculation result by the second processing unit; anda control unit configured to control the second processing unit and thedistribution information calculation unit on the basis of thecorrelation information calculated by the first processing unit.
 2. Thecontrol apparatus according to claim 1, wherein the control unitcalculates correlation information based on an image signal output fromone of the pixel portions positioned in the one of the second regionscorresponding to the one of the first regions, in which the correlationinformation calculated by the first processing unit varies, using thesecond processing unit, and calculates the distribution informationusing the distribution information calculation unit.
 3. The controlapparatus according to claim 2, wherein the control unit detectsvariation of the correlation information calculated by the firstprocessing unit on the basis of comparison between present correlationinformation and past correlation information, which are calculated usingan image signal output from one of the pixel portions positioned in theone of the first regions.
 4. The control apparatus according to claim 3,wherein the control unit controls the second processing unit tocalculate the correlation information using an image signal output fromone of the pixel portions positioned in the one of the second regionswhen an absolute value of differences between the present correlationinformation and the past correlation information, which are calculatedusing an image signal output from one of the pixel portions positionedin the one of the first regions, is larger than a threshold value. 5.The control apparatus according to claim 4, wherein the control unitcontrols the second processing unit to calculate the correlationinformation using an image signal output from one of the pixel portionspositioned in the one of the second regions when an absolute value ofthe differences based on image signals output from the pixel portionspositioned in the first regions is larger than a threshold value.
 6. Thecontrol apparatus according to claim 1, wherein the control unitcontrols the second processing unit to calculate the correlationinformation using an image signal output from one of the pixel portionspositioned in the one of the second regions when the correlationinformation, which is calculated using an image signal output from oneof the pixel portions positioned in the one of the first regions,indicates correlation lower than a reference.
 7. The control apparatusaccording to claim 6, wherein the control unit controls the secondprocessing unit to calculate the correlation information using an imagesignal output from one of the pixel portions positioned in the one ofthe second regions when the correlation information, which is based onimage signals output from the pixel portions positioned in the firstregions, indicates correlation lower than a reference.
 8. The controlapparatus according to claim 1, wherein the distribution information isinformation indicating distribution of an image shift amount betweenimage signals that respectively obtained from the photoelectricconversion portions of the one of the pixel portions.
 9. The controlapparatus according to claim 1, wherein the distribution information isinformation indicating distribution of a defocus amount between imagesignals that respectively obtained from the photoelectric conversionportions of the one of the pixel portions.
 10. The control apparatusaccording to claim 1, wherein the distribution information isinformation indicating distribution of an object distance based on animage shift amount between image signals that respectively obtained fromthe photoelectric conversion portions of the one of the pixel portions.11. The control apparatus according to claim 1, wherein, during settinga shooting mode that the focus detection unit generates the distributioninformation based on the image signal obtained by the image sensor, thefocus detection unit generates the distribution information using asignal output from the pixel portions positioned in the first regions.12. An image pickup apparatus comprising: an image sensor that has aplurality of pixel portions respectively including a plurality ofphotoelectric conversion portions and outputs an image signal from thephotoelectric conversion portions; a control apparatus including a firstprocessing unit configured to calculate correlation information ofbetween image signals, which are respectively output from one of thepixel portions positioned in one of a plurality of first regions, afocus detection unit configured to perform focus detection on the basisof a calculation result by the first processing unit, a secondprocessing unit configured to calculate correlation information betweenimage signals, which are respectively output from the one of the pixelportions positioned in one of a plurality of second regions, wherein theone of the second regions includes a region near the one of the firstregions, is larger than the one of the first regions, and corresponds tothe one of the first regions, a distribution information calculationunit configured to calculate distribution information corresponding toan object distance on the basis of a calculation result by the secondprocessing unit, and a control unit configured to control the secondprocessing unit and the distribution information calculation unit on thebasis of the correlation information calculated by the first processingunit; and a distance calculation unit configured to calculate the objectdistance on the basis of the correlation information calculated by thesecond processing unit.
 13. A control method of a control apparatus usedfor an image pickup apparatus including an image sensor that has aplurality of pixel portions respectively including a plurality ofphotoelectric conversion portions and outputs an image signal from thephotoelectric conversion portions, the control method comprising: afirst processing step of calculating correlation information betweenimage signals, which are respectively output from one of the pixelportions positioned in one of a plurality of first regions; a focusdetection step of performing focus detection on the basis of acalculation result by the first processing unit; a second processingstep of calculating correlation information between image signals, whichare respectively output from the one of the pixel portions positioned inone of a plurality of second regions, wherein the one of the secondregions includes a region near the one of the first regions, is largerthan the one of the first regions, and corresponds to the one of thefirst regions; a distribution information calculation step ofcalculating distribution information corresponding to an object distanceon the basis of a calculation result by the second processing unit; anda control unit step of controlling the second processing unit and thedistribution information calculation unit on the basis of thecorrelation information calculated by the first processing unit.
 14. Anon-transitory computer-readable medium configured to store an controlmethod of a control apparatus used for an image pickup apparatusincluding an image sensor that has a plurality of pixel portionsrespectively including a plurality of photoelectric conversion portionsand outputs an image signal from the photoelectric conversion portions,the control method comprising: a first processing step of calculatingcorrelation information between image signals, which are respectivelyoutput from one of the pixel portions positioned in one of a pluralityof first regions; a focus detection step of performing focus detectionon the basis of a calculation result by the first processing unit; asecond processing step of calculating correlation information betweenimage signals, which are respectively output from the one of the pixelportions positioned in one of a plurality of second regions, wherein theone of the second regions includes a region near the one of the firstregions, is larger than the one of the first regions, and corresponds tothe one of the first regions; a distribution information calculationstep of calculating distribution information corresponding to an objectdistance on the basis of a calculation result by the second processingunit; and a control unit step of controlling the second processing unitand the distribution information calculation unit on the basis of thecorrelation information calculated by the first processing unit.